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Image segmentation in time-series images 



5 The present invention relates to the field of digital imaging. In particular 

the present invention relates to a method of deternnning a first segmentation result of an 
object of interest in a first image of time-series images, to an image processing device 
and to a computer program for an image processing device. 

Segmentation methods are used to derive geometric models of, for 

10 example, organs or bones or other objects of interest from volumetric image data, such 
as CT, MR or US images. Such geometric models are required for a variety of medical 
applications, or generally in the field of pattern recognition. For medical or clinical 
applications, an important example is cardiac diagnosis, where geometric models of the 
ventricles and the myocard of the heart are required, for example, for perfusion analysis, 

15 wall motion analysis and computation of the ejection fraction. Another important 

clinical application is radio-therapy planning (RTP), where the segmentation of multiple 
organs and bones, for example, in the prostate region, is necessary for the diagnosis 
and/or the determination of the treatment parameters. 
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Deformable models are a very general class of methods for the 
segmentation of structures in 3D images. Deformable models are known, for example, 
from an article of T. Mclnerney et al "Deformable models in medical image analysis: A 
survey" in Medical Image Analysis, 1 (2): 91-108 1996. 

The basis principle of deformable models consists of the adaptation of 
flexible meshes, represented, for example, by triangles or simplexes, to the object of 
interest in an image. For this, the model is initially placed near or on the object of 
interest in the image. This may be done by a user. Then, coordinates of surface elements 
of the flexible mesh, such as triangles, are iteratively changed until they lie on or close 
30 to the surface of the object of interest. Such a method is described in further detail in J. 
Weese et al "Shape constrained deformable models for 3D medical image segmentation 
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"ZT C - fc ~ » >*— 1- Processing in Medical tanging 

(IPMO, page, 380 to 387, Davies, CA, USA, 2001, Springer Verlag. 

The optimal adaptation of an initial mesh is fonnd by energy 
mmnnizanon, where maintaining the shape of a geomehic mode, is traded offagains, 
-—-ft— P— of me objec, surfece in me image. Featore point detectionmay 
be earned on, iocally for each Mangle or simplex by searching for p ossi b,e object 
sorfaees m me inmge, for exatnple, along a normal of me Wangle or simplex 

Segmenting moving and/or deforming objects, snch aa, for example the 
.uniMhe bidder or me hear, from a ttae-aerrea of 3D images la difficul, in toe Z« 
s^rncan, snrface changes of me objec Becanse deformab.e models are loca. metoods 
men capnue range may be too small, resulting in segmentation errors 
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I, is an object of me presen, invention to provide for an improved 
segmentation of moving or deforming object* fhnn time-series of images 

Awrdta S tom ^'<>f«nepresen,inven n on,theaboveobjec.maybe 
:t2 am r 0fteennintogaflrS, ~ 0 — '.ofanobjectoLrereH 

- -end images. According to an aspect of me present invention, an 

-0 ^o^^meshtomeobjectofrntorestinmenrntimageisperformedto 
detemune me firs, segmentation reanlt The adaptation is perfomted on me basis of an 
energy optimization nsing me UU mesh ntd a shape model of the a* The 
mm., mesh corresponds to a second segmentation resnl. of me objec, of interest in me 
second -nage, which precedes me firs, image in the time-series images 

Advantageously, according to an aspec, of me presen, invention, a prior 
4D shape mode, is combined with me adaptation results o, a previons image. Dne to 
*rs, a memod ia provided which aUows to automatically mode, and segment stmchnes 
•ha, move and deform over time. Fnrmermore, advantogeous.y, toe presen, invention 
may allow to increase a robustness of me defotmable model segmentation for 4D 
appltcation, Fnrmemrom, me memod according to mis exempt embodiment of the 
presen. mvennon may allow to predict a nex, ttme step baaed on toe model and toe 
previous adaptation result. 
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According to another exemplary embodiment of the present invention as 
set forth in claim 2, an internal energy corresponding to a first distance between the first 
segmentation result and the shape model and an internal energy corresponding to a 
second distance between the object of interest and the first segmentation result are 
5 deterrnined, which are nnnimized. This allows for a fast and robust segmentation. 

Claims 3 to 6 provide for further advantageous exemplary embodiments 
of the present invention. 

According to another exemplary embodiment of the present invention as 
set forth in claim 7, an image processing device is provided suitably adapted for 

10 executing the method according to the present invention. Advantageously, this image 
processing device allows a very accurate and robust segmentation of moving and/or 
deforming objects in time-series images, where a failure may be avoided in case a 
change from one image to the next image is too large. Furthermore, improved 
segmentation results may be provided, since segmentation results from preceding 

1 5 images are incorporated in the determination of segmentation results in the actual 
image. 

According to another exemplary embodiment of the present invention, a 
computer program is provided allowing for an improved segmentation of moving or 
deforming objects in time-series images. The computer program may be written in any 

20 suitable programming language, such as C++ and may be stored on a computer readable 
device, such as a CD-ROM. However, the computer program according to the present 
invention may also be presented over a network such as the WorldWideWeb, from 
which it may be downloaded. 

It may be seen as the gist of an exemplary embodiment of the present 

25 invention that a prior 4D shaped model is combined with adaptation results of a 
previous image of the time-series images. According to an aspect of the present 
invention, the adaptation or segmentation result S(T) of the preceding image, is used as 
the initial mesh for the image I (T+l). For the adaptation, the model M (T+l) of the 
image I (T+l) is used as the corresponding shape model. In that way, the general a prior 

30 time varying shape model and the patient-specific image data (i.e. the previous images 
of the time sequences) are taken into account. 

These and other aspects of the present invention will become apparent 
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from and elucidated with reference to the embodiments described hereinafter. 

Exemplary embodiments of the present invention will be described in the 
following, with reference to the following drawings: - 



Fig. 1 shows a schematic representation of an image processing device 
according to an exemplary embodiment of the present invention, adapted 
to execute a method according to an exemplary embodiment of the 
present invention. 

Fig. 2 shows a simplified representation for further explaining the 
generation of a surface model, which may be applied in the method 
according to the present invention. 

Figs. 3a and 3b show a flowchart of an exemplary embodiment of t 
method for operating the image processing device of Fig. 1 according , 
15 the present invention. 

Fig. 4 shows a simplified representation for further explaining the 
present invention. 
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Fig. 5 shows asegmentation performed in accordance with an aspect of 
the present invention. 
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Fig. 1 shows a simplified schematic representation of an exemplary 
embodiment of an image processing device in accordance with the present invention. In 
Fig. 1 there is shown a central processing unit (CPU) or image processor 1 for adapting 
a deformable model surface to surfaces of an object of interest by mesh adaptation. The 
object may also be composed of multiple objects. In addition to being conceived to 
adapt a deformable model surface to the object surface, the image processing device 
depicted in Fig. 1 may also be adapted to determine or generate a surface model from 
one or a plurality of training models. 

The image processor 1 is connected to a memory 1 for storing the images 
of time-series images, i.e. a plurality of images which have been successively taken 
from a moving or deforming object. The image processor 1 may be connected by a bus 
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system 3 to a plurality of periphery devices or inpul/output devices which are not 
depicted in Fig. 1. For example, the image processor 1 maybe connected to an MR 
device, a CT device, an ultrasonic scanner, to a plotter or a printer or the like via the bus 
system 3. Furthermore, the image processor 1 is connected to a display such as a 
5 computer screen 4 for outputting segmentation results. Furthermore, a keyboard 5 is 
provided, connected to the image processor 1, by which a user or operator may interact 
with the image processor 1 or may input data necessary or desired for the segmentation 
process. 

Fig. 2 shows a simplified schematic diagram for explaining the 
1 0 generation of a surface model, which may be used in the method according to the 

present invention. In the following description, the present invention is described with 
reference to surface or shape models represented by triangular measures. However, it 
has to be noted that also simplex or polygonal meshes, or other suitable surface or shape 
models may be used. 

15 Reference numeral 10 designates first time-series images, which were 

taken of a first training model. The images respectively represent snap-shots of the 
moving or deforming training object at a certain time difference. Reference numerals 12 
and 14 designate further time-series images of further training models. Each of the time- 
series images 10, 12 and 14 consists of m 3D images I(t-0 ....m- 1), depicting the 

20 moving or deforming training object corresponding to the object of interest to be 
segmented later on at subsequent points in time t = 0 . . .m - 1 . 

Given N segmented time-series images I (t = 0 . . .m - 1), N * m triangular 
meshes may be derived in accordance with the method described in M.R. Kaus et al 
"Automated 3D PDM construction using deformable models" in 8 th International 

25 Conference on Computer Vision (ICCV), pages 566-572, Vancouver, Canada, 2001, 
IEEE Press, which is hereby incorporated by reference, such that for each segmented 3D 
image time-series, there is a set of m 3D triangular measures. Each mesh consists of V 
vertices with coordinates Vk, which are connected to W triangles. The topology of all 
measures is the same, i.e. V and W does not change. 

30 According to an aspect of the present invention, a shape model M (t), 

which is a prior 3D + shape model may be derived by calculating the mean coordinates 
from all N meshes of a time-point t. Thus, shape models M (0) . . .. M (m - 1) may be 
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generated for each imagel(t=0 ....m-l )o f the time-series images 10, 12 and 14. 

Thus, M (t) consists of a set of W triangles and V * m vert ex coordinates, 
i.e. each mesh has the same topology, and the vertex coordinates depend on the time, 
i.e. Vk(t) where 

5 k = 0- V- 1 andt = 0-m - 1. 

According to an aspect of the present invention, also additional 
information may be incorporated, such as, for example, an inter-individual variation of 
meshes at a particular time point t, using, for example, the principle component analysis 
as described by T.F.Cootes et al «A trainable method of parametric shape description" 
10 Image and Vision Comp., 1 0(5) :289-294, 1992, which is hereby incorporated by 

reference. Instead of the principle component analysis, other suitable representations 
may also be used. For example, it is also possible to interpolate between the time-points 
t to derive the vertex coordinates v k (t) without an explicit mesh M(t), such as a mesh at 
time (tO + (tl - tO) / 2). 

15 FigS - 3a 311(1 3b show a flowchart of an exemplary embodiment of a 

method for operating, for example, the image processing device depicted in Fig. 1 in 
accordance with the present invention which may be implemented as computer 
program. 

After the start in step SI, the method continues to step S2, where m time- 
20 series 3D images I (t) of a moving and/or deforming object of interest are acquired. In 
other words, a plurality of m images of a moving or deforming object of interest are 
read in, depicting the object of interest at subsequent points of time t. Then, in the 
subsequent step S3, a deformable model M (t) corresponding to the object of interest is 
read in. As mentioned above, the deformable model M (t) may have been determined as 
25 described with reference to Fig. 2. In the subsequent step 84, the first image I (0) of the 
time-series 3D images is loaded. 

In step S5, following step S4, an initial mesh is adapted to the object in 
the image I (0). The adaptation of the initial mesh to the object of interest is performed 
on the basis of an energy optimization using the initial mesh and the shape model M (0) 
30 of the image I (0) to determine a first segmentation result S (0) of the object of interest 
in the image I (0). 

This will now be described in further detail. After initial positioning of 
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the initial mesh, the initial mesh is adapted to the object of interest by iteratively 
carrying out a surface detection for the object surface of the object of interest in the 
image for each triangle and a reconfiguration of the vertex coordinates by rmnimizing 
the energy E = E„ t + a E mU wherein the parameter a weights a relative influence of the 
external energy Eext which drives the mesh towards detected surface points of the object 
of interest and the internal energy E int , which maintains the vertex configuration of the 
initial mesh, i.e. the form of the surface model M (0). 

The surface detection is carried out for each triangle center Xi of the 
initial mesh. A point x , is determined along a normal „ of the respective triangle which 
maximizes a cost function of a feature function F and a distance j8 to the triangle center 
according to 

x, = x, +8n, argmax{^(x, + j8n,)-Dj 2 5 2 } 

where 21 + 1 is the number of points investigated, 6 specifies the distance 
between two points on the profile, and D controls the tradeoff between feature strength 
and distance. A suitable feature function F may, for example, be taken from J. Weese et 
al "Shape constrained deformable models for 3D image segmentation" in proc 
IPMI'01, pages 380-387, 2001, which is hereby incorporated by reference. 

The external energy term drives the mesh towards the detected surface 

points: 

T 

- - = max{0,F(3f ( ) -Dj 2 6 2 }, 

with T being the number of triangles. The weights w, give the most 
promising surface points x, with the largest influence during mesh reconfiguration. 

The external energy maintains the distribution of the mesh vertex 

coordinates Vj , i.e. the edges of the initial mesh v- -v v 

v jk ~ v j~ v k 

v 

£ in.=Z S <. v j- v t-sXV Jlt ) 2 , 

J01 ksNU) 

where N(j) is the set of the neighbors of vertex j and V is the number of 
vertex coordinates. This is further described in J. Weese et al "Shape constrained 
deformable models for 3D medical image segmentation" in proc. IPMI'01, pages 380- 
387, 2001, which is hereby incorporated by reference. 

A rotation S and a scaling s of the mesh may be estimated for each 
iteration by using a fast closed-form point based registration method based on singular 
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value decomposition. Since the energies E„ t and are quadratic, an energy 
minimization results in an efficient solution of a sparse linear system using the 
conjugate gradient method. Then, after the adaptation of the initial mesh to the object of 
interest in the first image I (0) on the basis of a minimization of the external and internal 
energies, where the internal energy corresponds to distance between the segmentation 
result and the shape model and the external energy corresponds to distance between the 
object of interest and the segmentation result, tibe method continues to step S6, where a 
counter t is initialized with t = 0. The initial mesh used in step S5 may be a mean mesh 
of the surface model M (0). 

Then, in the subsequent step S7, the (t + l)th image I (t + 1) 0 f the time- 
series images is loaded. In other words, the subsequent image is loaded. Then in the 
subsequent step S8, the segmentation result S (t) of the preceding image I (t) is applied 
to the image I (t + 1) as the initial mesh. La other words, in case it is the first iteration 
with the counter t = 0, the segmentation result S (0) of the first image I (0) is used as the 
15 initial mesh for the adaptation in the subsequent image I (1). 

In the subsequent step S9, the initial mesh is adapted to the object of 
interest in the image S (t + 1) by using S (t) as the initial mesh and the shape model M (t 
+ 1). As described with respect to step S5, the adaptation may be performed on the basis 
of an energy minimization with respect to the internal energy Eiat and the external 
energy E^. The adaptation of the first mesh to the object of interest in the image I (t + 
1) may be performed in the same manner as described with respect to step S5, such that 
for a further explanation of the energy niinimization performed in step S9, it can be 
referred back to step S5. Then, when the energies have been minimized in step S9, i. 
cut-off criterion for the niinimization has been reached, the method continues to step 
S9, where it is determined whether the segmentation has been performed for all m 
images I (t) of the time-series 3D images. In case it is determined in step S9 that the 
segmentation has not been performed to all images of the time-series, the method 
continues to step SI 1, where the counter t is incremented t = t + 1 and the method 
returns to step S9. Then, the segmentation is performed for the subsequent image by 
using the segmentation results of the preceding image as the initial mesh and by using 
the shape model of the respective actual image as described above. 

In case it is determined in step S10 that the segmentation has been 
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performed for each image of the time-series, as indicated by the encircled A at the 
bottom of Fig. 3a and the encircled A at the top of Fig. 3b, the method continues to step 
S12, where the segmentation results S (0) to S (t) are output, for example, to the 
computer screen 4. Then, the method continues to step S13, where it ends. 

Hence, in accordance with an aspect of the present invention, for the 
segmentation of an object of interest in an image I ft) the surface model M ft) is used 
and the initial mesh for the segmentation process in the image I ft) is derived from the 
ammediately preceding image I ft - 1), which, as indicated above, may be the 
segmentation result S ft- 1). According to an aspect of the present invention, for the 
first image I (0), the mean mesh of the corresponding model M (0) may be used. 

Advantageously, the method according to the present invention may 
allow to automatically model and segment structures that move and deform over time 
with an improved accuracy. The present invention provides an increased robustness of 
the segmentation process, in particular for 4D applications. Furthermore, 
advantageously, the method may allow to predict the next time step based on the model 
and the previous adaptation result. 

The above described segmentation by deformable models may be used to 
derive geometric models for organs or bones from volumetric image data. 

Such geometric models in the above described segmentation may be in 
particular advantageous for a variety of clinical applications. For example, an 
advantageous application area may be the 4D radio therapy planning, where organ 
boundary delineation is necessary for the determination of the optimal treatment 
parameters and to evaluate a dose distribution in 4D. Another advantageous field of 
application may be cardiac diagnosis, where geometric models of ventricles and the 
myocard of the heart may be required for perfusion, wall motion and ejection fraction 
analysis. 

Fig. 4 shows a simplified drawing for further explaining an aspect of the 
present invention. As mentioned above, according to an exemplary embodiment of the 
present invention, a prior 4D shape model M (t) is combined with the adaptation results 
S (t - 1) of the previous images. As may be taken from Fig. 4, the shape model M (0) is 
used for the segmentation of the object of interest in the image I (0). The segmentation 
results of this first segmentation is S (0). As mentioned above, as the initial mesh for the 
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first image I (0), the mean mesh of the corresponding model M (0) may be used. Then, 
for the subsequent image I (1), the shape model M (1) is used and, as the initial shape 
for this second segmentation in the second image I (1), the segmentation result S (0) 
from the first segmentation is used. 
5 Hence, for the final segmentation of the final image I (m~l) of the time- 

series of images, the shape model M (m-1) is used and the segmentation result S (m - 2) 
of the subsequent segmentation as the initial mesh. 

Fig. 5 shows the segmentation process in four subsequent images I (2) to 
I (5) (in the upper line of Fig. 5) and the corresponding shape models M (2) to M (5) (in 
10 the lower line of Fig. 5). As can be gathered from the Figures, for each actual image, the 
corresponding shape model M (t) is used, but, as the initial mesh, the segmentation 
result S (t - 1) of the preceding image is used. Advantageously, this allows to perform 
an accurate segmentation of a moving or deforming object, even if the differences, i.e. 
the form or position differences, are large from one image to another. 



